Automatic Evaluation and Uniform Filter Cascades for Inducing N-Best Translation Lexicons

This paper shows how to induce an N-best t r ans la t ion lexicon f rom a bi l ingual t ex t corpus using s ta t is t ical proper t ies of the corpus toge the r wi th four ex te rna l knowledge sources. The knowledge sources are cast as filters, so t h a t any subset of t h e m can be cascaded in a un i form framework. A new object ive evaluat ion measure is used to compare the qual i ty of lexicons induced wi th different fi l ter cascades. The best f i l ter cascades improve lexicon qual i ty by up to 137% over the plain vanilla s ta t i s t ica l m e t h o d , and approach h u m a n performance . Dras t ica l ly reduc ing the size of the t ra in ing corpus has a much smaller impact on lexicon qual i ty when these knowledge sources are used. This makes it pract ical to t ra in on small hand-bu i l t corpora for language pairs where large bil ingual corpora are unavailable. Moreover , t h ree of the four filters prove useful even when used wi th large t ra in ing corpora.